feat: agent pipeline narration, run traces, verification structure fixes#20
Merged
Conversation
UX — the 6-step bar now tells what actually happened instead of bare 통과/실패: - Step summaries built from gate evidence (refs count, objective code, critic rounds/hints applied, SymPy solutions, re-solve answer + confidence, durations) - Emit the long-dead `preview` SSE event after generate (FE stage already existed) - Honest 3-state: `unverified` no longer masquerades as 통과 (△ badge), intent LLM fallback is labeled, new `attempt` event resets steps 3-6 with reason, new `runs` event surfaces parallel generation progress (n/m) DX — zero-logging agent now leaves a trail: - Per-run JSONL traces (request + every ProgressEvent incl. full gate evidence and critic/guard hints) under TRACE_DIR (default ./runs), plus llm.jsonl with per-call latency/tokens via a LanguageModel middleware; stdout narration per run Structure — fixes for flaws found in the D-5 review: - Split SOLVER_MODEL (gpt-5.4) so independent re-solve no longer shares the generator checkpoint (was sharing one gpt-5.5 instance, defeating D-5) - Generation failure degrades to a failed gate + skipped 4-6 and goes through the retry policy instead of killing the whole run (dead catch repaired) - RefinerAgent is now a real agent using prompts/refiner.md (repair-in-place, bounded change scope) instead of a disguised generator re-call - Critic critiques are preserved in gate evidence (critic_rounds, hints) - sympy_verify and re_solve run concurrently; driver errors are buffered so sibling runs can still deliver a result Wire contract (FE+BE updated together): step.status gains `unverified`, step.summary on success, new attempt/runs events; docs updated (D-6, web README). Tests: 247 passed (3 new test files), agent+web typecheck clean, live smoke verified against the running dev stack (preview/summaries/runs/traces observed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The SSE step sequence assertion encoded the old sequential contract; sympy_verify and re_solve now start together. Assert (index, status) pairs to document the parallel contract, and assert the preview event carries the formatLatex-formatted candidate text. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
무엇이 바뀌나
검증 화면이 "단계명 + 통과/실패"만 보여주던 것을, 각 에이전트가 실제로 한 일의 중계로 바꿉니다. 동시에 에이전트 서버에 관측성(JSONL 트레이스)을 깔고, D-5 멀티에이전트 구조 리뷰에서 확인된 결함 4건을 수리합니다.
UX — 스텝바가 서사를 말함
wire-adapter.ts개편)후보 생성 (gpt-5.5(xhigh)) · Critic 1라운드 · 지적 1건 반영 · 33.7초,SymPy 검산 일치 (해: -1, 4) · 1.5초preview이벤트를 BE가 emit — 3단계 직후 후보 문제가 화면에 뜸 (FE 렌더 경로는 이미 존재했음)unverified가 더 이상 ✓통과로 위장하지 않음 (△ "기호 검증 불가 — 재풀이로 확인"), intent LLM 폴백은 "(시드 의도로 대체)" 표기attempt이벤트: 재시도 시 "시도 n/m + 사유" 배너 + 3~6단계 리셋runs이벤트: 병렬 생성 런 집계 (문항 n/m 완료) + 첫 런 종료 후 대기 구간 안내DX — 로그 0줄 → run 단위 트레이스
TRACE_DIR(기본./runs)에 run별 JSONL: 요청 전문 + 모든 ProgressEvent(게이트 evidence, Critic/가드 힌트 전문 포함)llm.jsonl: LanguageModel 미들웨어로 모든 LLM 호출의 지연/토큰/finish_reason 계측 (에이전트 코드 무수정)구조 수정 (D-5 검토 결과)
SOLVER_MODEL=gpt-5.4분리 (llm.jsonl에서 작동 확인)prompts/refiner.md(v0.2.0, 변경 범위 제한)를 쓰는 진짜 수선 에이전트로 교체critic_rounds/critic_hints_total을 evidence에 보존 → summary/트레이스로 노출sympy_verify∥re_solve병렬화Wire 계약 변경 (FE+BE 동시 수정)
step.status에unverified추가, 성공 시에도summary제공,attempt/runs이벤트 신설.docs/specs/architecture.mdD-6과packages/web/README.md§SSE Consumption 갱신.검증
refiner-agent,verification-workflow-generate-failure, wire 서사/preview 케이스)tsc --noEmit클린gpt-5.4 (solver)호출 전부 관측됨후속 거리 (이 PR 범위 밖)
objective-mapping.ts,rag-search.ts)의 strategy 데이터 파일 이동🤖 Generated with Claude Code
Summary by cubic
Adds narrated step summaries, early candidate preview,
attempt/runsSSE events, and per‑run traces so reviewers can see what happened in each pipeline step. Also separates the solver model, hardens retries on generation failure, and replaces the refiner with a real repair agent.New Features
unverifiedis now shown (△).previewright after generate so the first candidate appears earlier.attempt(retries with reason; resets steps 3–6) andruns(parallel run progress n/m).TRACE_DIRwith allProgressEventevidence;llm.jsonllogs latency/tokens via a LanguageModel middleware; concise stdout narration per run.unverified/attempt/runs/previewand show notices; docs updated (docs/specs/architecture.md,packages/web/README.md). Integration test asserts parallel 4/5 and preview formatting.Bug Fixes
SOLVER_MODEL(defaultgpt-5.4) instead of sharing the generator checkpoint.generation_failed; withlast-resort, fall back to the deterministic template.prompts/refiner.md(bounded, in‑place repair with one schema‑retry path).critic_roundsand total hints in gate evidence; shown in summaries and traces.sympy_verifyandre_solverun in parallel; driver errors are buffered so sibling runs can still return results.Written for commit dacde49. Summary will update on new commits.